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ABSTRACT 



This thesis examines the relationships between independent 
variables such as underway time, material condition, and 
personnel manning and dependent variables in the form of 
engineering exam scores. Multinomial linear regression models 
are used to examine these relationships. These efforts met 
with limited success. The percent of time that a ship spends 
underway prior to an OPPE was the most significant of any 
independent variable considered, yet efforts to model the 
effects of diminishing returns were unsuccessful. Outchop 
OPPEs failed to show any significant relationship for the 
underway independent variables examined, but they did reveal 
that ships which file a greater number of CASREPs prior to 
receiving an outchop OPPE increase their odds of receiving a 
favorable test score. Attempts to model LOE were 
unsuccessful . 
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I. INTRODUCTION 

A. BACKGROUND 

The ability of a Navy ship to perform its peacetime and 
wartime missions is a function of its various departments' 
(operations, weapons, engineering, navigation, and supply) 
abilities to conduct their individual tasks. The primary 
goals of the engineering department, providing propulsion, 
generating electricity, and controlling battle damage, are an 
intricate part of the ability of the ship as a whole to 
perform its mission. Consequently, the measurement of a 
ship's engineering readiness plays a role in the overall 
assessment of the ship's ability to complete its mission. 
To evaluate the engineering department of each conventionally 
powered surface ship, the Commander in Chief, Atlantic Fleet, 
established the Propulsion Examining Board. 

The Atlantic Fleet Propulsion Examining Board (PEB) 
consists of Naval officers with shipboard engineering 
experience. Headed by a post-command Captain, the board is 
divided into several inspection teams. Each examination team 
consists of a post-command Commander, as well as other 
officers with engineering experience at the department head 
and division officer levels. The head of the PEB is tasked 
with inspecting Atlantic Fleet ships using a uniform standard 
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of engineering readiness. Since the board operates out of one 
office, with one clear set of standards, exam results may be 
viewed as a good aggregate measure of a ship's engineering 
readiness. The PEB conducts two basic types of examinations. 
Operational Propulsion Plant Examinations (OPPE's) and Light 
Off Examinations (LOE's). 

The primary purpose of an LOE is to determine the ship's 
ability to safely operate the engineering plant equipment. 
Additionally, an LOE evaluates the ability of a ship to combat 
a flammable liquid fire in a main engineering space. LOE's 
are typically administered after an idle period of 120 days 
or more of the ship's engineering plant. Usually, such a 
period of inactivity occurs during a period of maintenance 
such as a regular overhaul (ROH) or selected restricted 
availability (SRA) . An LOE is also administered prior to the 
commissioning of a ship. 

On the other hand, OPPE's are administered to a ship in 
an operating status. The PEB conducts an OPPE for each ship 
approximately every 24 months. While an OPPE also evaluates 
the ship's abilities to safely operate propulsion plant 
equipment and to combat a flammable liquid fire, the focus of 
the exam is somewhat different. The PEB examines the ability 
of the ship to man a three section underway watchbill. It 
also determines the ability of the ship to conduct its own 
underway training and evaluation using in-house evaluation 
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teams known as the Engineering Casualty Control Training Team 
(ECCTT) and the Damage Control Training Team (DCTT) . 

The OPPE and LOE are the best overall measures of a ship's 
engineering readiness. The examination criteria are the most 
comprehensive of any exam administered to surface units. The 
PEB compares current and past ship engineering procedures 
against prescribed fleet and type commander requirements. 
The PEB assigns qualitative grades to the exam results for 
both exams. Exams do not receive quantitative scores on a 
numeric scale, for example, zero to 100. Each exam receives 
an overall score which is a reflection of the results of 
various subareas such as the ship's material status, level of 
knowledge, training, firefighting, ability to operate 
equipment safely, and the like. The LOE's are evaluated on 
a pass-fail basis, while OPPE's receive relative scores such 
as above average, average, below average, and unsatisfactory. 

The material evaluation of the ship's engineering plant 
includes both a static and dynamic inspection. Prior to any 
other major portion of the inspection, the PEB evaluates the 
physical condition of the equipment. The crew provides an 
initial assessment which establishes that a specified minimum 
level of equipment is functioning. The PEB then requires 
selected equipment to be demonstrated as operational; for 
example, a qualified operator may be required to start a fire 
pump correctly and place it in operation. An evolution such 
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as this also helps PEB to assess the level of knowledge of the 
engineering department. 

The PEB evaluates the level of knowledge of the department 
through the successful operation of plant equipment, as 
previously discussed. It also utilizes oral and written 
exams. The written exams arrive at the ship just prior to the 
exam. Critical watchstanders also undergo a formal board of 
oral examination. In addition, crew members are also subject 
to questioning by PEB members at any point during the exam. 

The engineering department's operating logs, legal, and 
administrative records are also inspected for proper format 
and recordkeeping. The administrative review helps to 
establish the ship's ability to implement required, safety- 
related programs. It also inspects the ability of the ship 
to conduct requisite watchstation training and follow up with 
the appropriate documentation. The review in these areas 
focuses on the ability of the ship to schedule personnel to 
attend required schools, to conduct required organizational 
training onboard, and to correctly record these events in the 
required manner. The PEB also inspects the ship's operating 
and legal records. This inspection provides additional 
insight into the material status of the equipment. It also 
demonstrates the ability of the crew to identify problem 
areas, for instance an out of tolerance temperature reading, 
and to take corrective action. However, the operating log 
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inspection is but one technique available to PEB for the 
evaluation of proper casualty control procedures. 

Perhaps the most intense period of the exam occurs during 
the graded casualty control drills. During these drills, the 
ship's own casualty control training teams, ECCTT and DCTT, 
conduct casualty drills on various equipment. The PEB 
evaluates the ability of the shipboard team to impose, 
conduct, and evaluate these drills, as well as the ability of 
the crew to effectively handle the casualty. The PEB members 
constantly question and evaluate every level of watchstander - 
- from the commanding officer through the engineering officer 
of the watch to the lowest ranking fireman on duty in the 
engineering spaces — to determine each individual's current 
assessment of the casualty, their required actions, and 
subsequent possible effects. The majority of the drills focus 
solely on the engineering department, but one area of 
evaluation, firefighting, concerns all hands. 

The PEB's assessment of firefighting focuses on the 
ability of the ship to combat a major fire in a main 
engineering space. A typical scenario involves the discovery 
of a major flammable liquid leak in a main engineering space. 
As the drill advances, the casualty becomes more complex, 
usually progressing into a main space fire. While engineering 
department personnel would typically combat such a fire, 
additional personnel are tasked to provide backup and 
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support. The PEB will typically evaluate the qualifications 
and performance levels of these personnel as well. The main 
space fire drill is one of the most difficult phases of the 
entire exam. 

The PEB administers the most comprehensive engineering 
examinations for surface ships. While other measures of 
readiness, such as casualty reporting procedures, may provide 
a more detailed static assessment, PEB provides the only 
uniform inspecting team which evaluates the current status and 
inspects the past operating procedures in an attempt to ensure 
safe engineering plant operation in the future. 

B . PURPOSE 

Since PEB administers exams on a fleet wide basis, 
examination results are of great interest to the other levels 
of the chain of command, such as squadron, group, and type 
commander, in addition to the ships themselves. While all 
levels strive for high engineering readiness as a goal, each 
attempts to specifically aid or prepare a ship for an upcoming 
exam. Perhaps the best example is that of the Commander Naval 
Surface Force Atlantic (CNSL) Engineering Mobile Training Team 
(EMTT) . This team conducts periodic and preliminary 
inspection visits on ships in an effort to make them better 
prepared for an OPPE or an LOE. The squadron or type 
commander may also schedule an additional maintenance period, 
or additional days underway for training, in the hope of 
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achieving a higher score. In fact, CNSL maintains a shipboard 
training and readiness division, N6, which is tasked with 
monitoring ship engineering readiness and assisting ships in 
their preparation for engineering inspections. 

Both the type commander and the squadron commander support 
a ship's efforts to increase its score on an OPPE or LOE. 
Both staffs contain personnel intimately familiar with 
shipboard engineering requirements through their own 
experiences at the senior enlisted and officer level. With 
such a high level of interest focused on this one measure of 
effectiveness, the examination process becomes a natural topic 
for academic study. The Center for Naval Analysis (CNA) has 
conducted the majority of previous work is this area. 

Linda Cavalluzzo, a CNA analyst working at the recpjest of 
CNSL, performed the initial study to interpret possible 
critical inputs into the exam scores [Ref. 1] . It focuses on 
the effect of an increased number of days underway for CNSL 
ships receiving an OPPE. Cavalluzzo uses a pass/fail 
criterion as her measure of effectiveness (MOE) . While her 
results are promising (i.e., they indicate an increased chance 
of receiving a passing grade for those ships with a greater 
number of underway days) , her study is limited to a small 
sample of ships (22) over a short period of time. The brevity 
of the analysis and small number of observations are no doubt 
the fallout of a timely requirement for the results. 



7 



r 



Alan Marcus et al. also attempt to evaluate the 
relationships between OPTEMPO, the number of days a ship 
spends underway, and readiness [Ref. 2]. The study is more 
comprehensive in nature, covering a greater number of ships 
(134) over a longer period of time (1982-1985) . The study 
uses the results of a selected exercise (SELEX) in navigation, 
the ability of individual shipboard departments to win 
competitive awards, as well as OPPE results as the primary 
MOEs. Marcus also chooses the pass/fail criterion as the 
primary OPPE MOE, thus combining a possible four or five 
choices into two distinct categories. He also focuses the 
majority of his analysis on one critical input factor — 
OPTEMPO. Marcus utilizes a similar rationale for review as 
did Cavalluzzo, yet he concludes that the pass rate for an 
OPPE had no association with OPTEMPO. This sets the stage for 
the current analysis. 

This thesis builds on the work of Marcus and Cavalluzzo 
in an attempt to discover a resources-to-readiness 
relationship between various critical input factors and their 
effect on OPPE and LOE exam scores. Previous work has focused 
primarily on OPTEMPO as an important factor. This thesis will 
consider a broader scope of inputs, such as various 
measurements of material readiness as well as the level of 
manning. It will also evaluate the level of detail provided 
by the exam results — resisting the temptation to utilize a 
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simple pass/fail criterion — and will rely instead on the 
greater qualitative distinction provided by the exam scores. 

By broadening the scope of study in the number of exams 
reviewed, possible input factors considered, and the 
assessment of output measures, this thesis seeks to determine 
relationships that decision makers may use in assessing 
various input factors and their effects on ship exam scores. 
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II. DATA AND SOURCES 



A. INPUT DATA 

As with the previous studies this effort utilizes data 
from a CNA database. The database contains data on 152 
surface ships from fiscal year FY 1978 through FY 1988. The 
variables reflect monthly measures of their respective 
quantities, with the exception of several manpower variables, 
which due to CNA computation reflect 90 or 180 day values. 
The data originate from a number of different sources, most 
of which are primary monitoring or decision making tools for 
the Navy or Department of Defense. Initially compiled in 
response to a CNA effort concerning the factors affecting ship 
material condition, the variables cover a wide range of 
topics. Reference 3 describes these sources in depth. In 
part, the sources include the following: 

- Ship Employment History (SEH) 

- Officer Master Files 

- Defense Manpower Data Center's Enlisted Personnel 
Unit Identification Code File 

- Enlisted Master Record Files 

- Enlisted Billet Files 

- Board of Inspection and Survey 

- 3-M Maintenance Data System 

- 3-M Parts Records 
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- Casualty Reporting System (CASREPs) 

- Ship Fuel and Steaming Hours Data 

In general, the data fall into one of three basic 
categories based on the quantities which they describe: 
material condition of the ship, manpower, or underway time. 

The CNA database contains several useful measures of each 
ship's underway time. The variables which describe the 
percent underway and the percent in port each month are the 
most simple and direct measure. When they are added together, 
they sum to 100.00. Another variable contains the number of 
hours steaming underway in a month. Its companion contains 
the number of hours steaming not underway in a month. This 
measure allows analysis in the scenario where a ship operates 
the engineering plant tied to the pier or at anchorage. Other 
underway variables include the percent deployed, the percent 
long deployment (i.e., when a ship is on duty in the Western 
Pacific, Indian Ocean, or Persian Gulf) , and the percent of 
extended operations (greater than eight weeks away from 
homeport) . 

Several related measures of manpower are also present in 
the database. Manpower variables divide easily into two 
different types: those which describe the current level of 
manning relative to the prescribed level and those which 
measure the rate of crew turnover. 



11 



The measures of crew turnover are for 90 (one quarter) and 
180 day (two quarters, or six months) periods. Values 
available include the percent new crew overall, which applies 
to all enlisted personnel job specialties, or ratings, and the 
percent new crew in the engineering job ratings. 

The variables which describe the manning level relative 
to requirements also display values for both the entire crew 
and engineering ratings. The CNA data use two weights in the 
computation of these ratios. The first weight utilizes only 
the difference in pay among enlisted personnel. The second 
attempts to capture differences in the productivity of 
enlisted personnel based on their paygrade. The weighting 
scheme is the result of CNA follow up work on a Rand 
Corporation study. Appendix D of Reference 3 contains 
additional details. The numerator values of these ratios, the 
manning levels, are also available for study. 

Manning values are subdivided into three paygrade groups. 
Manning to requirements measures exist for paygrades of junior 
personnel (E1-E3) , mid-grade petty officers (E4-E6) , and 
senior petty officers (E7-E9) for the overall crew. A measure 
of this ratio is also available for mid-grade engineering 
petty officers (E4-E6) . 

The final input variables available for study are those 
which describe the material condition of the ship. All of 
these measures utilize Casualty Reporting (CASREP) data. A 
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CASREP is a message sent from the Commanding Officer of a ship 
to his administrative and operational commanders describing 
equipment which is not functioning as designed. These 
messages document discrepancies which cannot be repaired 
within 48 hours after discovery. Three categories exist to 
describe the degree of degradation, C2, C3, and C4 , in 
increasing order of seriousness. 

All of the data present describing material readiness 
prior to exams in the CNA database utilize measures derived 
from CASREPs. All values measure monthly events. The first 
variable measures the number of CASREPs which begin in a given 
month by degree of degradation. One value also exists which 
describes the number of C3 and C4 CASREPs which begin in a 
given month; it is the sum of the individual measures. A 
variable also exists which describes the number of days a ship 
spends in a C3 status in the month. For example, if a ship 
has two C3 CASREPs for a 30 day period, this value would be 
60. Again, an aggregate measure exists for the C3 and C4 
status combined. Finally, variables exist to describe the 
number of days that a ship is free of CASREPs. These values 
range from 0 to 31, with the exception of the aggregate 
measure. Days Free of C3/C4 CASREPs. This measure has values 
in the range from 0 to 62. 

At least one caveat is important to note in the material 
measures at this juncture. The CASREP data, while accurately 
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describing the material condition of the ship, includes 
casualties from other areas of the ship. Consequently, a 
casualty from the combat systems department will affect 
variable values, yet may be insignificant in the ability of 
the ship to pass an engineering exam. 

One shortfall exists in the overall database, a result of 
the past focus of CNA studies. In general, auxiliary ships 
(such as ammunition ships, oilers, and food carrying ships) 
and amphibious ships (such as helicopter assault ships and 
tank landing ships) lack a complete set of input variables for 
study. This is the result of CNA studies which center on 
combatants. However, due to several CNA projects concerning 
manpower, the majority of the manpower-related input variables 
are present for all ships. The initial effect of this 
discovery requires more careful procedures when processing the 
data, and the ultimate impact is to restrict the range of 
ships to which possible conclusions apply. 

In summary, the CNA database contains accurate measures 
which have previously been used for modeling and analysis. 
While the data were initially used for a CNA study, they have 
been expanded and updated to cover additional areas of 
interest. 
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B . OUTPUT DATA 



Output data are primary source data. They come directly 
from the PEB. The data describe the results of every type of 
exam administered by the PEB. As previously discussed, the 
PEB gives two basic types of exams, OPPEs and LOEs. Their 
database distinguishes some differences among these two types, 
however. For example, the results differentiate between the 
exams and reexams. Since the majority of reexaminations occur 
shortly after the initial evaluation date, the input variables 
are not noticeably different. Also, the PEB usually 
reinspects only those failing areas of the initial exam. For 
these reasons, reexams are not reviewed in this study. It 
also distinguishes between regular OPPEs and OPPEs given on 
the way home from a major deployment. The latter type of 
exam is called an outchop OPPE. The data also break out exams 
administered by the Atlantic Fleet Training Group (FTG) in 
Guantonomo Bay, Cuba. Since this specific type of exam is no 
longer administered, it is not considered in this review. 

Basically, the PEB data consist of the exam results, both 
the overall score and every subarea. It also contains 
ancillary information such as the ship's homeport, 
administrative squadron assignment, etc. As previously 
discussed, the PEB assigns qualitative grades such as 
unsatisfactory, below average, average, and above average to 
OPPE exam results, and evaluates LOEs on a pass/ fail basis. 
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The PEB also records incomplete exams and the portions of 
exams which are not given. This policy results in a 
comprehensive data collection mechanism. While the majority 
of the PEB data concern output measures, three input variables 
are also present. 

The PEB database denotes the experience level of the 
engineering officer by showing the number of months which he 
has been in his current job. It reflects the experience level 
of the Commanding Officer (CO) in the same manner. The PEB 
also records if the CO was an engineer when he served as a 
department head. All of these factors contribute to the level 
of managerial expertise on board prior to the exam. 

In summary, the PEB database contains the exam results 
collected by the examination teams. With a Commander 
verifying the results prior to release, and one officer 
managing the database, the data are very accurate. The PEB 
data were matched to the CNA data and uploaded into the Naval 
Postgraduate School (NPS) mainframe computer for processing. 
A discussion of the methodology follows. 
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III. METHODOLOGY 



As stated in the introduction, this thesis examines the 
resource-to-readiness relationships between input factors, or 
independent variables, and engineering exam scores, or 
dependent variables. Traditional approaches which search for 
a relationship between independent and dependent variables 
often utilize simple linear regression. However, due to the 
structure of the problem under study, and the assumptions of 
linear regression, this technique is inappropriate. A related 
procedure, called logistic regression, is the effective 
analytical tool for the job. The following section will 
demonstrate the suitability of logistic regression, after 
describing the shortfalls of linear regression. 

Consider a simple linear regression model of an exam which 
assigns pass/fail scores. Linear regression uses the equation 

yj = a + bXj + e. 

for i = 1,...,N, where y- is a random variable which takes on 
the value of 1 if ship i passes the exam and 0 if it fails, 
a and b are unknown parameters, x- is the input factor under 
consideration, and e,. is a random disturbance which is assumed 
to be normally distributed with a mean of zero and a variance 
of for all i. Let P,- be the probability that y- = 1, and 
let (1 - Pj) be the probability that yj = 0. Note that if 
E[e, ] = 0, then E[yj] = P,- = a + bx,- . This approach 
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immediately generates a number of problems. Note that while 
Yj is restricted to 0 or 1, the systematic portion of the 
right hand side of the equation may take on any value. 
Consequently, e- can have only two values, -(a + bXj) and 1- 
(a + bXj) . It assumes these values with probabilities of 1 - 
(a + bXj) and -(a + bx-), respectively. Since a + bx,- can have 
values greater than one or less than zero, P- can also be less 
than 0 or greater than 1. These glaring contradictions are 
further compounded by the variance of e- . The traditional 
assumption that Var(e,) = for all i no longer applies. 
Instead, the Bernoulli character of y,- requires that the 
variance for e- become (a + bx^) (1 - a - bxJ . Notice that 
the variance of e- now depends on i. This violates of one of 
the basic assumptions of simple linear regression that the 
variance of e- is the same for all i. Finally, computations 
for E(yjxj) may result in values outside of the specified 
range of [0,1]. Judge et al. [Ref. 5] discuss these and other 
shortfalls of using linear regression to model this scenario. 

Since the simple linear model fails to correctly depict 
the basic pass/fail exam outcome, the binary logit is 
considered [Ref. 6]. The binary logit model utilizes a linear 
regression model of the following form 

S 

Yj* = Bq + E B.x.j + u,. 

D=1 
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where y- is not observed. It is called a "latent" variable. 
The value actually observed is a dummy variable where 

1 if y* > 0 
0 otherwise. 



y,- = 



The term x,.j represents the value of the independent 

variable for the i^'^ observation. The vector Bj, where 
j=l,...,S, is a vector of coefficients assigning various 
weights to the input variables. Note that in this model, 
unlike the previous one, E[yjxj] is not Bq + BjXjj, rather 
these terms represent the expected value of the latent 
variable y^. The previous two relationships allow us to state 

S 

Pj = Prob(y,. = 1 ) = Prob[ u, > - ( B^ + ) ] 

j = l 



S 



= 1 - F[ -( Bq + E B.x.j) ] 

j=l 



where F is the cumulative distribution of u. Using the 
assumption that the distribution of u is symmetric (i.e., 1- 
F(-Z) = F(Z)), the equation takes the form 



S 

Pi = F( B„ + EBiX,. ). 

3-1 

The functional form of this equation depends on the 
assumptions concerning the error term u. If the cumulative 
distribution of u,- is logistic, then 
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F(Zj) 



(Zj) 

1 + exp (Z,.) 



S 

where Z- = Bg + 12 

j=l 






Further, we obtain 



F(Z^) 

log = Z-, 

1 - F(Zj) 



and in terms of the model under discussion 



P. S 

log = Bg + L B.Xj.. 

1 - P. j=l 



The left-hand side of the equation, known as the log-odds 
ratio, describes the effects of increasing or decreasing the 
explanatory variables on the logarithm of the odds of success. 
Consequently, the binary choice logit model has the final 
form 

S 

exp( Bg + Z B.Xj. ) 

j = l 

P. = 

S 

1 + exp( Bg + Z Bj.Xj. ) 

j=l 



This form is adequate for the pass/fail case. Clearly, a more 
complex model is required for the case where more than one 
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discrete output is possible. An extension of the previous 
model is an appropriate choice. 

The OPPE results consist of a qualitative response in the 
form of discrete scores. A Rand Corporation study [Ref. 7], 
discusses such a structure, where the response variables are 
categorical. Let y. denote an assigned score for ship i, 
where i = 1,...,N. Then let 

Pij* = P{Yi = aj} 

where the response variable, exam score, for the i^^ ship can 

o * • 

assume Q values, a^,...,ag. Note that P.j = 1. This 

requires that the i ship must, in fact, receive one of the 

possible scores. The response variables are related to the 

input, or stimulus, variables by the standardized multivariate 

logistic CDF, which is defined by 



1 

F(ti, . . . , t < tj < 00. 

n 

1 + exp( -tj ) 

j=l 



Defining tj in terms of z, and applying this distribution to 
the proposed model yields 

exp( z.j ) 






, 1“1,...,N/ ^ 1,...,Q< 



k=l 



exp( Zj,^ ) 
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Define z,j = xJBj, where x,. is a S x 1 vector of input 
characteristics for ship i. Bj is the vector, S x 1 of the 
coefficient weights to be estimated. The coefficients may be 
estimated via a weighted least squares technique or a maximum 
likelihood estimation (MLE) technique. Until recently, the 
former was utilized. This is primarily the result of the time 
required to implement the latter on a computer; however, the 
software package which supports this thesis. Time Series 
Processor (TSP) , uses MLE techniques. 

Once the computer generates an estimate for the 
coefficient, it is time to assess its significance. A "t- 
test" is conducted to determine if the given estimate is 
significantly different from zero, implying a relationship 
between the input factor and exam score. Since the 
coefficient estimates are MLE ' s and the estimated standard 
errors are the square roots of the diagonal elements of the 
asymptotic variance-covariance matrix, which is derived from 
the Cramer-Rao lower bound, this is an asymptotic t-test. The 
value of the t-statistic is the estimate of the coefficient 
divided by the standard error [Ref. 6]. In most cases, the 
sign of the coefficient is known prior to computation. For 
instance, an increase in the number of days free of CASREPs 
would have a positive impact on the exam score. Consequently, 
a one sided "t-test" is performed; otherwise, when the sign 
of the estimated coefficient is unknown, a two-sided test is 



22 



used. Throughout the course of analysis, the actual value of 
the t-statistic will be presented to avoid disputes concerning 
the appropriate level of significance. 
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IV. PROPOSED MODELS 



As discussed during the review of the data, available 
input factors fall into one of three broad categories: 
manpower, underway time, or material condition. The data 
collection techniques used in assembling the database result 
in the variables having a high degree of collinearity within 
each category. Consider, for example, those variables 
measuring a ship's material condition. All of the measures 
are based on the CASREP system. The number of days a ship 
spends in a C3 status is simply the number of days in the 
month minus the number of days it spends free of C3 CASREPs. 
In a similar manner, the number of CASREPs which begin in a 
month's time influences the number of days a ship spends in 
a casualty status. Consequently, it is important to determine 
which variable within each category makes the greatest impact 
on the exam score. 

The simplest way to approximate which variable has the 
greatest significance is to conduct several univariate runs. 
The sign of the estimated coefficient is not the critical 
issue, since it is usually known prior to the run. The 
magnitude of the estimate and the associated significance is 
what counts. A value of zero for the coefficient would imply 
no relationship between the independent and dependent 
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variables. TSP presents the estimated coefficient, standard 
error, and t-statistic for review on the print out. The value 
of the t-statistic is the key to evaluating the importance of 
the estimate. 

Once the most significant variable in each category is 
determined, multivariate models may be approximated using the 
most significant variables from the univariate analysis. 

The OPPE scores are a function of the underway time, manpower 
status, and material condition. These values are used by TSP 
in the x vector, as discussed in the previous section. The 
B vector contains the values of the estimated coefficients. 

The LOE model uses the multinomial logit formulation as 
well, since the multinomial reduces to the binomial case when 
there are only two possible choices. The LOE scores are a 
function of both the manpower status and material condition 
of the test ship. Since LOE ' s occur after a ship has not 
been underway for at least 120 days, underway time is not 
considered for this model. The ability of a ship's material 
status to influence LOE scores also becomes a concern at this 
point. Recall that all material measures are based on the 
CASREP system. When a ship enters an overhaul period, the 
reporting procedures for casualties change. This allows 
shipyard workers to repair and replace equipment without a 
CASREP being filed. In view of this new concern about the 
ability of material variables taken in combination with 
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earlier remarks concerning the utility of the material 
variable as a whole, the value of this measure is in some 
doubt. But since overhaul periods comprise only one type of 
event which precedes an LOE, the CASREP measure may still have 
some validity. 
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V. 



ANALYSIS 



A. UNIVARIATE MODELS 

TSP presents estimated coefficients for review in tabular 
form. Since the software normalizes the estimated 
coefficients, the value for the lowest coefficient is set to 
zero. For the LOE, where the exams are reported as a pass or 
fail, the effect is to estimate the coefficients reflecting 
the constant term and the log odds of passing the exam. For 
the OPPE, the coefficient for the log odds of receiving a 
failing score is set to zero. Since the focus of this thesis 
is to determine important input factors which result in ships 
successfully completing these exams, this development is not 
a significant shortcoming. 

Recall that the majority of the data are available in 
monthly values. Runs were conducted varying the number of 
months in the independent value from one to six. Usually, 
when the three months or six months previous to the month in 
which the exam was given were used as the input value, the 
most significant results were achieved. Consequently, one or 
two quarters prior to the month of the exam is used most often 
for the independent variable values. This is also in keeping 
with the planning and scheduling timeframes utilized by the 
Navy. 
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1. Material Variables 



As previously discussed, univariate runs were 
conducted using the LOE, OPPE, and outchop OPPE as output 
measures across all three variable types to determine which 
variable from each of the three categories is the most 
significant. Material variables were considered first. When 
examining the effects of CASREP status on OPPE scores, it is 
logical to assume that those values which measure more severe 
degradations would have a greater impact on the examination 
score. Univariate runs using material measures of readiness 
support this assumption. The Days Free of C3/C4 CASREPs in 
the quarter prior to the month of the exam (DF34-90) is the 
most significant material variable for the regular OPPE. As 
shown in Table 1, all of the signs of the slope coefficients 
are positive as anticipated. This implies that more days free 
of C3/C4 CASREPs increase the log odds of receiving a passing 
score (i.e., below average, average, or above average). The 
estimated slope coefficient for those ships receiving above 
average scores is greater than the other estimates. Its t- 
statistic value is also the greatest, giving it a level of 
significance of less than 0.05. This implies that there is 
less than a five percent chance of error when stating that of 
the ships receiving an OPPE, an increase in the number of days 
free of C3/C4 CASREPs increases the log odds of attaining that 
score. The other estimated coefficients have a level of 
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TABLE 1. OPPE vs. DF34-90 (Degrees of Freedom = 277) 



Parameter 



Estimate 



T-Statistic 



Below Average 

Constant 

DF34-90 

Average 

Constant 

DF34-90 

Above Average 

Constant 

DF34-90 



-0.7512428 

0.006690118 



-0.1199808 

0.006795538 



-1.083841 

0.6529236 



-0.2061219 

0.7839620 



-2.511097 

1.789323* 



-3 . 120854 
0.02949016 



♦significant at the .05 level 



significance greater than 0.20, but they are not significant 
at the 0.05 level. 

Ships receive outchop OPPEs at the end of the 
deployment enroute to homeport. A ship has usually been away 
from homeport for a period of approximately six months. While 
the Navy endeavors to give increased support to deployed 
units, the maintenance patterns differ from a non-deployed 
status. For instance, parts support is more difficult. While 
stores ships carry spare parts for deployed ships, their 
capacities are limited. Also, when a ship is away from its 
homeport on deployment, it does not have access to the local 
Navy Supply Center or local civilian suppliers. Consequently, 
shipboard patterns for CASREPs change during a deployment, 
perhaps in an attempt to prompt the supply system for needed 
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parts. This causes the most important material measure to be 
different for outchop OPPEs than for regular OPPEs. The most 
significant material measure affecting outchop OPPE scores is 
the number of C4 CASREPs which begin in each month for the six 
months prior to the month of the exam (BC4-180) . As shown in 
Table 2, the sign of the estimated slope coefficient is 
certainly not as expected. In short, it says that ships which 

TABLE 2. OUTCHOP OPPE VS. BC4-180 
(Degrees of Freedom = 72) 



Parameter 



Estimate 



T-Statistic 



Below Average 

Constant -1.4572926 

BC4-180 0.4040139 



-3.097871 

2.455777** 



Average 

Constant 

BC4-180 



-0.6774675 

0.3070861 



-1.820998 

2.001085* 



Above Average 

Constant -1.788364 

BC4-180 0.4261859 



-3.392287 

2.482655** 



* Significant at the 0.05 level 
** Significant at the 0.025 level 



file more C4 CASREPS in the six months prior to the month of 
the exam have an increase in the log odds of receiving a 
passing score. This shipboard strategy is not unexpected in 
general, since it is sometimes employed in an attempt to 
expedite parts or assistance prior to the start of a 
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deployment; however, it is surprising that it surfaces in this 
instance which occurs near the end of a deployment. The t- 
statistic value is significant at the 0.05 level, and the 
magnitude of the estimated coefficients are the greatest seen 
thus far. 

Material measures also influence the LOE scores. 
Recall that ships receiving an LOE either pass or fail the 
exam. The LOEs are affected most by the number of days of 
C3/C4 CASREPs in the two quarters prior to the month of the 
exam (CD34-180) , as shown in Table 3. The sign of the 
coefficient is negative as anticipated, indicating that an 
increase in the number of days of C3/C4 CASREPs has a negative 
TABLE 3. LOE VS. CD34-180 (Degrees of Freedom = 96) 

Parameter Estimate T-Statistic 

Passing 

Constant 2.794936 1.265409 

CD34-180 -0.05296375 -1.204150* 

* Significant at the 0.10 level 



impact on achieving a passing score. The magnitude of the 
estimated coefficient and its level of significance are less 
than for previous input factors considered. 

2 . Underway Time Variables 

Those variables which measure the amount of time a 
ship spends underway are the next category for discussion. 
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Recall that since an LOE is administered after a prolonged 
inport status, it is not discussed here. As shown in Table 
4, the variable which measures the percent of time a ship is 
underway in the six months prior to the month of the exam 
(PRUW-180) has the greatest impact on the OPPE exam scores. 
Unlike the CASREP variables, which were the most significant 
for the quarter before the exam, it appears that underway time 
TABLE 4. OPPE VS. PRUW-180 (Degrees of Freedom = 277) 



Parameter 



Below Average 

Constant 

PRUW-180 

Average 

Constant 

PRUW-180 

Above Average 

Constant 

PRUW-180 



Estimate 



-1.430346 

0.006204640 

-1.186394 

0.008172133 

-3.429899 

0.01122343 



T-Statistic 



-2.143820 

1.847222* 



-2 . 029214 
2.762979** 



-3.416571 

2.493490** 



* Significant at the 0.05 level 
** Significant at the 0.01 level 



has the most impact for the six months, or two quarters, prior 
to the exam. Again, the signs of the estimated slope 
coefficients are positive as anticipated, implying that a 
higher percentage of time underway is beneficial in the six 
months prior to the month in which the exam is given. The t- 
statistic values for the estimated coefficients are all 
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significant. The least significant has a level of 
significance of less than 0.05. 

The percent of time spent underway in the three months 
prior to the exam (PRUW-90) is the most significant underway 
measure for outchop OPPEs. The signs of the estimated slope 
coefficients in Table 5 are positive, as anticipated. Both 
the coefficient and the t-statistic value are the greatest for 
the average test score, implying that an increase in the 
percent of time underway had the greatest impact in achieving 
this score. The lowest t-statistic value occurs for the above 
average score, and with a level of significance of less than 
0 . 10 . 

TABLE 5 . OUTCHOP OPPE VS. PRUW-90 (Degrees of Freedom = 71 ) 



Parameter 



Below Average 

Constant 

PRUW-90 

Average 

Constant 

PRUW-90 

Above Average 

Constant 

PRUW-90 



Estimate 



-1.4017210 

0.006307510 



-0.94444166 

0.006826114 



-1.561182 

0.005502330 



T-Statistic 



-2.443562 

1.597810* 



-1.954103 

2.014881** 



-2.541660 

1.288903* 



* Significant at the 0.10 level 

** Significant at the 0.05 level 
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3 . Crew Manning Variables 

Variables which reflect crew manning also play a role 
in influencing exam scores. The percent of turnover in 
enlisted personnel in the engineering department in the 
quarter prior to the month of the exam is the most important 
factor for OPPEs (PNE3-90) . As shown in Table 6, the signs 
of the estimated slope coefficients are negative as expected, 
TABLE 6. OPPE VS. PNE3-90 (Degrees of Freedom = 277) 



Parameter 



Below Average 

Constant 

PNE3-90 

Average 

Constant 

PNE3-90 

Above Average 

Constant 

PNE3-90 



Estimate 



-0.2919169 

-0.005819893 



0.6977273 

-0.01693883 



-1.063960 

-0.01160226 



T-Statistic 



-0.8072801 

-0.5342568 



2.295294 

-1.783363* 



-2.117330 

-0.7318880 



* Significant at the 0.05 level 



implying that an increase in the rate of personnel turnover 
in the engineering department decreases the log odds of 
receiving that score. Only the estimated slope coefficient 
for an average test score is significant at the 0.05 level. 

Variables which describe the crew manning for the 
months prior to an outchop OPPE produce some interesting 
results. Both the percent of new engineering department 
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personnel in the quarter prior to the month of the exam (PNE3- 
90) , and the level of manning of midgrade petty officers 
shipwide for the quarter prior to the exam (M46R-90) impact 
on outchop OPPE scores. They are both presented here to 
indicate their relatively equivalent level of significance and 
to justify the variable chosen for incorporation in the 
multivariate outchop OPPE model. 

The PNE3-90 has a significant impact on the scores as 
shown in Table 7 ; however, the sign is the opposite of that 

TABLE 7. Outchop OPPE VS. PNE3-90 (Degrees of Freedom = 71) 



Parameter 



Below Average 

Constant 

PNE3-90 

Average 

Constant 

PNE3-90 

Above Average 

Constant 

PNE3-90 



* 

** 



Estimate 



-2.818458 

0.08362997 



-1.492497 

0.05407537 



-3.282754 

0.09071160 



Significant at the 
Significant at the 



T-Statistic 



-2.522146 

2.048857* 



-1.719659 

1.591466* 



-2.627789 

2.037764** 



0.10 level 
0.05 level 



which was expected. It is positive, implying that an 
increased rate of personnel turnover prior to an exam is 
beneficial. This is a counterintuitive result. No known 
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cause would produce this outcome. Like CASREP patterns, 
personnel turnover patterns are different during a deployment. 
Shipboard personnel tend to transfer at a greater rate before 
or after a deployment vice during a deployment. So while the 
values of the t-statistics are significant for all estimated 
coefficients at a minimum of the 0.10 level, the resulting 
reversal in the signs of the estimated slope coefficients make 
PNE3-90 inappropriate for consideration for the multivariate 
model . 

The M46R-90 also plays a significant role in 
influencing the scores for outchop OPPEs as shown in Table 8. 
The signs of the estimated coefficients are positive as 

TABLE 8. OUTCHOP OPPE VS. M46R-90 
(Degrees of Freedom = 71 ) 



Parameter 



Estimate 



T-Statistic 



Below Average 

Constant -9.7692376 

M46R-90 0.03492278 



-2 . 611423 
2.468589** 



Average 

Constant -5.330087 

M46R-90 0.02021113 



-1.881447 

1.825406* 



Above Average 

Constant -7.000989 

M46R-90 0.02357663 



-1.872073 

1.639826* 



* Significant at the 0.05 level 
** Significant at the 0.025 level 
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anticipated, implying that a higher level of manning is 
better. The t-statistic values indicate that all the 
estimated coefficients are significantly different from zero 
at less than the 0.05 level. Since this measure of manning 
influence is more in line with expectations, and since no 
apparent explanation for the sign reversal can be found for 
the percent turnover measure, the former is chosen for 
inclusion in the multivariate model. 

Personnel manning also affects the outcome of LOEs. 
Both the amount of engineering crew turnover in the six months 
prior to the month of the exam (PNE6-180) and the level of 
manning shipwide for the junior enlisted personnel, E1-E3, for 
the month prior to the month of the exam (M13R-90) impact LOE 
scores. They are both presented in discussion here to review 
their relative level of significance and to justify the one 
selected for inclusion in the multivariate model. 

Table 9 displays the effects of PNE6-180 on LOE 
scores. The sign of the estimated coefficient is negative, as 
anticipated, implying that a higher rate of crew turnover 
decreases the odds of receiving a passing score. 

The M13R-90 also has an impact on the log odds of 
achieving a passing LOE score, as shown in Table 10. The sign 
of the estimated slope coefficient is positive, implying that 
a higher rate of manning is better. It is interesting to note 
that the t-statistic value is greater for M13R-90 than for 
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TABLE 9. LOE VS. PNE6-180 (Degrees of Freedom = 96) 



Parameter 


Estimate 


T-Statistic 


Passing 

Constant 

PNE6-180 


1.863670 

-0.004528640 


4.386748 

-1.733043* 


* 


Significant at 


the 0.05 level 


TABLE 10. LOE 


vs. M13R-90 (Degrees of Freedom = 97) 


Parameter 


Estimate 


T-Statistic 


Passing 

Constant 

M13R-90 


-1.286832 

0.02682330 


-1.153997 

2.223840* 


* 


Significant at 


the 0.025 level 



PNE6-180, even though the input variable is only for the month 
prior to the month of the exam vice the six months prior. The 
significance level for M13R-90 is less than 0.025. The 
absolute magnitude of the estimated slope coefficient is also 
much greater for M13R-90 than for PNE6-180, even though the 
latter measures the rate of crew turnover specifically in the 
engineering department. For these reasons, M13R-30 is the 
measure incorporated into the multivariate LOE model . 
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B. MULTIVARIATE MODELS 

Multivariate models are formed using the most significant 
variables from the univariate runs. The goal is to use those 
measures which have already demonstrated a relationship to 
capture the interactions which may occur among the input 
variables. These attempts met with little success. The 
multivariate model for the OPPE uses DF34-90, PRUW-180, and 

PNE3-90, in Table 11. Like the univariate models, the 

TABLE 11. OPPE vs. DF34-90/PRUW-180/PNE3-90 
(Degrees of Freedom = 134) 



Parameter 



Below Average 

Constant 

DF34-90 

PRUW-180 

PNE3-90 

Average 

Constant 

DF34-90 

PRUW-180 

PNE3-90 

Above Average 

Constant 

DF34-90 

PRUW-180 

PNE3-90 



Estimate 



-1.459331 

-0.0009838671 

0.006435732 

0.0003021655 



-0.7600887 

-0.001248508 

0.008181062 

-0.01382492 



-5.146638 

0.02929889 

0.01007018 

0.0000682056 



T-Statistic 



-1.873792 

-0.1187338 

1.876532* 

0.2523762 



-1.044490 

-0.1738605 

2.744023*** 

-1.111305 



-3.563135 

1.885787* 

2.183477** 

0.04564056 



* Significant at the 0.05 level 

** Significant at the 0.025 level 

*** significant at the 0.01 level 
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value is so low as to make them insignificant. The importance 
of percent time underway is underscored by this model . In 
fact, it clearly dominates the other input factors. Yet, the 
estimated slope coefficients are not very great in magnitude. 

Attempts to form a significant multivariate model for 
outchop OPPEs met with similar results as shown in Table 12. 
The model uses BC4-180, PRUW-180, and M46R-90 as independent 
variables. Like the univariate manpower model for outchop 
OPPEs which also uses M46R-90, the significance of the 
estimated slope coefficients for the manpower measures are low 
with the exception of the below average score. The signs of 
the estimated coefficients are positive. This implies that 
an increase in the level of manning has a positive, albeit 
small, impact on below average exam scores. Also, note the 
relative insignificance of the percent time underway as an 
input factor. This is the opposite of the regular OPPE, where 
it was important. Perhaps this is a result of the outchop 
OPPE being given at the end of a deployment where all ships 
would normally have a higher operating tempo, reducing the 
impact of individual differences on exam scores. The 
magnitude of the estimated coefficients is higher than 
previous models for BC4-180, indicating, as in the univariate 
outchop OPPE model, that ships which file more casualty 
reports in the six months prior to the exam increase the log 
odds of achieving a passing score. 
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TABLE 12 



OUTCROP OPPE VS. BC4-180/PRUW-180/M46R-90 
(Degrees of Freedom = 47) 



Parameter 


Estimate 


P-Value 


Below Average 


Constant 


-13.37920 


-1.864744 


BC4-180 


0.4134871 


1.946976* 


PRUW-180 


-0.002257835 


-0.3016684 


M46R-90 


0.04615580 


1.779415* 


Average 


Constant 


-4.081422 


-0.7678036 


BC4-180 


0.2322206 


1.229982 


PRUW-180 


0.004349979 


0.6688791 


M46R-90 


0.007969758 


0.3853083 


Above Average 


Constant 


0.2036105 


0.0325641 


BC4-180 


0.3560169 


1.674598* 


PRUW-180 


-0.009437287 


-1.141737 


M46R-90 


0.00463012 


0.1889376 



* Significant at the 0.05 level 



The multivariate LOE model met with poor results as shown 
in Table 13. It uses CD34-180 and M13R-30 as independent 
variables . 

TABLE 13. LOE VS. CD34-180/M13R-90 
(Degrees of Freedom = 96) 



Parameter 



Estimate 



T-Statistic 



Passing 

Constant 

CD34-180 

M13R-30 



0.7387126 

-0.002938007 

0.001935358 



0.7810106 

-0.3680441 

0.5478099 
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Again, the motivation for the selection of these variables 
and their respective times rests with the univariate model. 
The model calculates the signs of the estimated slope 
coefficients as negative and positive respectively. This 
implies that fewer C3/C4 CASREPs and junior enlisted higher 
manning are better. Yet, the value of the t-statistics are 
so low as to make them insignificant. 
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VI. CONCLUSIONS 



This thesis set out to confirm relationships between input 
measures of manpower, underway time, and material readiness, 
and their impact on LOE and OPPE exam scores. By broadening 
the measure of performance to reflect the level of detail 
recorded by the PEB, the models attempt to capture these 
effects via multinomial logistic regression using a software 
package called TSP. These efforts met with limited success. 

The analysis effort involved using independent variables 
from the CNA ship database. The majority of ships considered 
had complete manpower variables a result of past CNA study 
efforts. However, other independent variable categories, such 
as material and underway time measures, are complete only for 
combatant ships such as cruisers, destroyers, and frigate. 
This shortfall immediately restricts the scope of ships to 
which any possible conclusions may apply. The database could 
be improved by incorporating the missing values for other ship 
types . 

The percent of time that a ship spends underway was the 
most significant of any independent variable considered. The 
models presented demonstrated for the OPPE that an increase 
in the amount of time that a ship spends underway influences 
the log odds of achieving a desired test score in a positive 
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way. However, it would be inappropriate for decisions makers 
to opt to keep ships underway all the time in an attempt to 
get an above average score. Clearly, a ship must spend some 
amount of time pierside for maintenance and crew rest. This 
concept implies that the effects of underway time should be 
a quadratic instead of a linear function (i.e., more underway 
time is desirable only to some point and then the ships 
experience diminishing returns) . Attempts to fit a quadratic 
function in support of this idea failed. Future work should 
focus in this area since it remains an important issue. 
Additionally, the models failed to show a strong relationship 
between outchop OPPEs and underway time, perhaps due to the 
fact that these ships spend a higher percentage of time 
underway as a group resulting in smaller differences in amount 
of underway times among ships. 

Material measures affected the exam scores, but to a 
lesser degree. The most interesting discovery was that the 
number of C4 CASREPs which begin in the six months prior to 
an outchop OPPE has a positive effect on achieving a desired 
score. Again, perhaps, this is a result of an attempt by 
ship's force personnel to prompt the supply system to expedite 
required parts. Material measures have even less impact on 
LOE scores, due in part to the fact that when a ship enters 
an extended maintenance period, the casualty reporting rules 
are relaxed. 
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As previously noted, CASREP variables, as they appear in 
the database, reflect degradations throughout the entire ship. 
Consequently, equipment failures in departments other than 
engineering inflate all the CASREP variables considered. 
Since casualty reports specify the equipment failure and 
cognizant department, the CASREP variables can be categorized 
by department. CNA already breaks down manpower variables by 
department in the database in an attempt to measure events 
more precisely. Future work examining the impact of material 
status engineering scores would benefit by having the same 
level of detail available for material measures. 

Manpower measures also affected exam scores, but again to 
a lesser degree than anticipated. The models constructed 
indicate that a reduced rate of crew turnover and a higher 
level of crew manning is desirable in the months prior to the 
exam. In general, the rate of crew turnover was a more 
significant influence than the level of manning of the ship. 

Other factors not examined may be influencing the exam 
results. For instance, the type or age of the propulsion 
plant may also influence the exam results. Perhaps the time 
of the year also affects the exam outcome. These and other 
possible input factors are areas for future research. 

The focus of the analysis was to comfirm existing 
hypotheses between various input factors and exam scores. 
This thesis does not try to measure how well the models 
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presented would perform in predicting exam performance. Such 
an effort would require reliance upon some sort of correlation 
coefficient, or r^. An unambiguous, uniformly-agreed-to r^ 
measure has not been developed for multinomial logit models. 

In summary, the thesis confirmed relationships between 
independent input factors and dependent exam scores. The 
relationships were significant to a lesser degree than 
anticipated. The most prominent was the positive relationship 
between underway time and the log odds of achieving a 
successful test score. Yet, attempts to capture the effects 
of diminishing marginal returns were not satisfactory. 
Further work is needed to model these effects. 
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